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VEHICLE CONTROL SYSTEM, PROGRAM AND METHOD 

TECHNICAL FIELD 

The present invention relates to a vehicle control system, program and 
5 method for detecting the approach of another vehicle based on images captured 
by a camera mounted on a primary vehicle and instructing the primary vehicle to 
take some action depending on detection result. 

BACKGROUND OF THE INVENTION 

10 Conventionally, a sensor or a camera mounted on an automobile is used to 

detect an approaching vehicle based on its output. In most cases, presence / 
absence of the approaching vehicle is detected by pattern matching between 
features extracted from captured images and templates prepared in advance. 
For example, Japanese Patent No.3014233 discloses an approaching object 

15 detecting apparatus. According to the apparatus, a captured image is divided 
into a plurality of window regions. A pattern corresponding to the image is 
created with features extracted from each window region as a data array. By 
comparing the pattern with the captured image, the approaching object 
apparatus may be detected automatically and reliably. 

20 Various methods may be used to extract features from the captured image. 

Among those, many methods use optical flows. For example, Japanese Patent 
No. 3011566 discloses an approaching vehicle monitoring apparatus. According 
to the apparatus, optical flows are computed based on captured images. By 
detecting a moving object and its movement direction through the use of the 
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optical flows, only approaching vehicle may be detected separately from its 
background. Since optical flows are used to eliminate noise according to this 
invention, more accurate detection may be realized. 

There are some techniques for not only detecting the approaching vehicle 
5 by image processing but controlling the vehicle by utilizing the result of the 
image processing. For example, Japanese Unexamined Patent Publication 
(Kokai) No. 2001- 101592 discloses a vehicle collision prevention apparatus. By 
detecting conditions of an intersection before entering it, the apparatus takes an 
appropriate action to avoid colUsion. If it is determined that collision avoidance 
10 action is necessary, the apparatus instructs the vehicle to make a braking or 
temporarily prohibit the driver's pressing of an accelerator pedal. 

In order to detect the approach of another vehicle based on an image, 
pattern matching is necessary between the image and templates. However, if 
the templates should be prepared in advance, a large number of templates are 
15 required to make an appropriate vehicle control in many different situations. 
This may cause a problem of storage capacity. 

There may be much difference between scenes on which the driver 
determines that collision avoidance action is needed depending on some 
information such as road conditions, weather and the driver's personal character. 
20 Therefore, if the number of templates for determination is limited, flexibihty is 
lost and collision avoidance action can be made in inappropriate condition, which 
makes a bad effect on driving. 

Therefore, there is a need for vehicle control system which is flexibly 
applicable to driver's characteristics and various situations around the primary 
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vehicle with relatively small storage capacity. 

SUMMARY OF THE INVENTION 

According to the present invention, every time a driver presses a brake 
5 pedal, an image captured in a traveling direction of a primary vehicle is reflected 
to a template. When an image similar to the template is captured and the 
driver does not perform a collision avoidance action such as releasing an 
accelerator and pressing a brake pedal, a brake system throttle valve is forcibly 
closed. Thus, the accelerator becomes out of control and sufficient braking force 

10 is provided even if harsh braking is performed by the driver. 

One aspect of the invention provides a vehicle control system for detecting 
the approach of another vehicle to the vehicle mounting the control system 
(hereinafter referred to as a "primary vehicle") to execute a collision avoidance 
action. The vehicle control system comprises capturing means for capturing an 

15 external image from the primary vehicle and template memory means for storing 
templates for detecting the approach of another vehicle. The system further 
comprises template update means for updating the templates when a brake pedal 
is pressed by a driver and recognizing means for comparing the external image 
with the template and calculating an evaluation value to determine whether the 

20 another vehicle approaches the primary vehicle based on the result of said 
comparing. The system further comprises instruction means for instructing the 
primary vehicle to perform the collision avoidance action when said evaluation 
value exceeds a threshold value. 

According to such aspect of the invention, based on the image captured in 
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the traveling direction, of the primary vehicle, the control system determines 
whether it is necessary to decelerate or stop the primary vehicle because of short 
distance between vehicles by comparing the evaluation value with a threshold 
value. Then, when it is determined that the primary vehicle needs to be 
5 decelerated or stopped and when the driver does not perform the collision 
avoidance action such as releasing an accelerator or pressing a brake pedal, a 
control signal for instructing the primary vehicle to perform the colhsion 
avoidance action is sent to the primary vehicle. As a result, collision between 
vehicles may be prevented. The number of threshold values ("ki" in a preferred 

10 embodiment) is not limited to one. The type of collision avoidance action may be 
changed depending on which threshold value the evaluation value exceeds. 
Note that the collision avoidance action includes forcibly closing a throttle valve 
of a brake system. Thus, the vehicle driving force is decreased and a sufficient 
negative pressure is provided to a brake booster. As a result, even when the 

15 driver's judgment is delayed and harsh braking is required, sufficient braking 
force may be provided to the vehicle. The collision avoidance action includes 
making an alarm or a warning display for the driver, forcibly releasing an 
accelerator pedal and forcibly pressing a brake pedal. 

The template is composed of a plurality of sub-regions and each sub-region 

20 has an eigen value. The eigen values should be, but not exclusively, pixel 
intensity values in 256 -level gray scale. The values may be calculated by 
applying a Fourier transform, a Gabor transform or a wavelet transform. 

The recognizing means calculates the evaluation value according to the 
following procedure. The captured image and the template are compared for 
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each sub-region to calculate a similarity factor of each sub-region. Optical flows 
are calculated based on the image. The similarity factor and the optical flows 
are multiplied to calculate a feature value of each sub-region. Then, by adding 
all of the feature values and weights assigned to sub -regions, the evaluation 
5 value is calculated. The feature value is, for example, a local movement Vn in 
the preferred embodiment described below. Further, the similarity factor is 
calculated by comparing an arithmetic mean value of the pixel intensity values 
contained in each sub-region with the eigen value of each sub-region. 

When the brake pedal is pressed by the driver, the template update means 

10 increases the weight of the sub-region containing approach information of 
another vehicle (see the process performed on a weighting value Wn at steps S308, 
S312 in the preferred embodiment). The template update means operates to 
bring the eigen value of a given sub-region closer to the value for the 
corresponding sub -region in the image (see the process at steps S308, 312 in the 

15 preferred embodiment). Thus, the conditions under which the driver judges the 
need for braking may be stored as templates and personal character of the driver 
and traffic conditions are reflected on the determination of the approach of the 
vehicle. 

Preferably, templates are stored in a template memory means as a 
20 template set. The template set is composed of templates with a positive weight 
and templates with a negative weight. The template set is not limited to one set. 
Three types of template (that is, for traveling straight, turning left and turning 
right) may be prepared depending on traveling direction of the primary vehicle. 
In such a case, the template update means updates the template in the template 
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set corresponding to the traveling direction of the primary vehicle. The 
recognizing means calculates the evaluation value with respect to the template in 
the template set corresponding to the traveUng direction of the primary vehicle. 
Thus, since templates are updated every time the brake pedal is pressed by the 
driver instead of preparing the large number of fixed templates, storage capacity 
for templates may be relatively small. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram of a brake system, which is an object to be controlled by 
a vehicle control system of the invention; 

Fig. 2 is a functional block diagram of the vehicle control system; 

Fig. 3 is a diagram showing an example of a position where a camera is 
installed in a vehicle; 

Fig. 4 is a diagram showing a construction of a template; 

Fig. 5 is an example of an image captured by the camera; 

Fig. 6 is another example of the image captured by the camera; 

Fig. 7 is a diagram showing a template prepared in advance; 

Fig. 8 is a flowchart of vehicle approach recognizing process performed by 
an image processing unit; 

Fig. 9 is a diagram showing similarity factors assigned to each sub-region 

of the template; and 

Fig. 10 is a flowchart of template updating process performed by the image 

processing unit. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Fig. 1 shows an example of a brake system 1 being controlled by a vehicle 
control system according to the invention. The brake system 1 comprises a 
brake pedal 2 and an accelerator 4 provided in a cabin of a vehicle. A sensor is 
5 provided to each of these pedals for detecting whether the pedal is being pressed. 
The brake pedal 2 is connected to a brake booster 6. A negative pressure 
generated inside an intake manifold 10 of an internal combustion engine 8 is 
supplied to the brake booster 6 via a pressure conducting pipe 12. Using the 
negative pressure, the brake booster 6 generates assisting force according to the 
10 pedaling force of the brake pedal 2. 

A master cylinder 14 connected to the brake booster 6 generates master 
cylinder pressure Pm depending on the sum of the brake pedahng force and the 
brake assisting force of the brake booster 6. The master cylinder pressure Pm is 
conducted to hydrauUc systems 16, 18, 20, 22 provided corresponding to four tires 
15 FR, FL, RR, RL, where braking power is produced to each tire. 

Wheel cylinders 24, 26 providing braking power to the tires FR and FL are 
connected to normally opened solenoid valves 28, 30 and normally closed solenoid 
valves 32, 34, respectively. Wheel cylinders 36, 38 providing braking power to 
the tires RR and RL are connected to normally opened solenoid valves 40, 42 and 
20 normally closed solenoid valves 44, 46, respectively 

The master cylinder 14 is of a tandem type and includes an output port 48 
connecting to the hydrauUc systems 16, 18 and an output port 50 connecting to 
the hydraulic systems 20, 22. 

The brake system 1 further includes emission valves 52, 54, intake valves 
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56, 58, a motor 60, pumps 62, 64, auxiliary reservoirs 66, 68, and proportional 
pressure reducing valves 70, 72. The proportional pressure reducing valves 70, 
72 are equipped for balancing the brake control by proportionally reducing the 
braking power apphed to the rear wheels (tires RR, RL) relative to the front 
5 wheels (tires FR, FL) according to reduction amount of rear wheel load on 
braking. 

The normally open solenoid valves 28, 30, 40, 42, the normally closed 
solenoid valves 32, 34, 44, 46, the motor 60 and the proportional pressure 
reducing valves 70, 72 are controlled by an electronic control unit (ECU) 74. 

10 When the brake pedal is pressed by a driver, the normally opened solenoid 

valves 28, 30, 40, 42 are opened, the normally closed solenoid valves 32, 34, 44, 
46 are closed and the pumps 62, 64 are stopped in response to a signal from the 
ECU 74. As a result, the master cylinder 14 and the wheel cyUnders 24, 26, 36, 
38 become connected. The master cylinder pressure Pm is supplied to the wheel 

15 cylinders 24, 26, 36, 38 and thus the braking force corresponding to the driver's 
pedaUng force is generated. 

During antilock control, the pumps 62, 64 become connected in response to 
a signal from the ECU 74. The normally opened solenoid valves 28, 30, 40, 42 
and the normally closed solenoid valves 32, 34, 44, 46 are closed or opened as 

20 required. Accordingly, the pressure in the wheel cylinders 24, 26, 36, 38 is 
increased, decreased, or maintained as required. For example, when it is 
detected that the tire FR is about to be locked up unnecessarily, the normally 
opened solenoid valve 28 is energized to be closed and the normally closed 
solenoid valve 32 is energized to be opened. As a result, some of the brake fluid 
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pressure in the hydraulic system 16 flows into the auxiliary reservoir 68 and so 
the braking force on the tire FR is reduced. The fluid stored in the auxiUary 
reservoir 68 is returned back to the hydraulic system 16 by the pump 64. 

Depending on depression of the accelerator pedal 4, the ECU 74 controls 
5 opening of a throttle valve 76 of the internal combustion engine 8 over a range 
from totally closed to full throttle. As the degree of opening of the throttle valve 
76 increases, rotation speed of the internal combustion engine 8 increases, 
thereby increasing the driving force of the primary vehicle. 

The greater the opening degree of the throttle valve 76 is, less negative 
10 pressure is generated in the intake manifold 10. Therefore, the negative 
pressure supplied from the pressure conductor 12 to the brake booster 6 is 
decreased. As a result, the assisting limit pressure from the 
brake booster 6 is reduced. When harsh braking is performed while the 
assisting limit pressure is reduced, the master cylinder pressure Pm exceeds the 
15 assisting limit pressure and vehicle braking force corresponding to the 
pedal-pressing force may be varied compared to normal braking situation. 

One objective of the invention is to generate sufficient vehicle braking force 
even when harsh braking occurs. The vehicle control system according to one 
embodiment of the invention determines whether the primary vehicle needs to be 
20 decelerated or stopped based on the image captured in the traveling direction 
when the distance between vehicles becomes short. When the primary vehicle 
needs to be decelerated or stopped and the driver fails to perform a collision 
avoidance action such as releasing the accelerator or pressing the brake pedal, 
the vehicle control system controls the throttle valve 76 of the brake system 1 to 



Q02186US 



be closed. Thus, the vehicle driving force decreases and sufficient negative 
pressure is supplied to the brake booster 6. Therefore, even when harsh braking 
is required because of the delay of driver's judgment, sufficient braking force may 
be generated. 

« 

5 Fig. 2 is a functional block diagram of the vehicle control system 80 

according to one embodiment of the invention. The vehicle control system 80 
includes a camera 102 for capturing an image, an image processing unit 100 for 
determining the need of a control instruction for the primary vehicle based on the 
image, and the ECU 74 for sending control signals to some parts of the brake 

10 system 1 according to the control instruction. 

The camera 102 (for example, a CCD camera) is mounted at the front of 
the primary vehicle and captures images in the vehicle traveling direction at 
regular intervals. Captured images are sequentially transmitted to the image 
processing unit 100. Fig. 3 shows an example of mounting position of the 

15 camera 102. The camera 102 may be installed in other positions. Two or more 
cameras may be used. The image is a digital image having pixels with intensity 
values providing 256-level (firom 0 to 255) gray scale. Alternatively, an image 
with pixels providing greater or less level gray scale may be used. 

The image processing unit 100 includes an image memory section 104, a 

20 recognizing section 106, an instruction section 108, a control signal receiving 
section 110, a template update section 114, and a template memory section 116. 
These functional blocks of the image processing section 100 may be implemented 
in software or hardware. The image processing section 100 may be implemented 
as some part of the ECU 74. 
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The image memory section 104 stores some sequential images received 
from the camera 102. For example, the section 104 stores two sequential 
images. 

The recognizing section 106 determines to send control signals to the ECU 
5 74 based on the images stored in the image memory section 104. Specifically, 
the recognizing section 106 processes the images stored in the image memory 
section 104 and a template Mn stored in the template memory section 116 
(detailed process will be described later) to recognizes situation in the traveling 
direction of the primary vehicle. When the vehicle moves forward, the section 
10 106 determines whether another vehicle traveling ahead approaches the primary 
vehicle. If so, the section 106 further determines whether some control is 
needed on the brake system 1. Determination result is sent to the instruction 
section 108. 

The instruction section 108 sends appropriate control instruction to the 
15 ECU 74 according to the determination result of the recognizing section 106. 

The ECU 74 then executes some control such as closing the throttle valve 76. 

In the template memory section 116, templates are stored for matching 

with images. Assume that the number of stored templates is represented as "n" 

(n=l, 2...) and each template is indicated as Mn. 
20 Fig. 4 shows one example of template layout. One template includes 400 

sub-regions in 20 columns by 20 rows. Each sub-region in the template 

corresponds to pixels in a sub-region having same size in the captured image. 

Each sub -region has one eigen value. In this embodiment, the pixel intensity 

value is used as the eigen value. This eigen value is updated based on the image 
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captured by the camera 102 every time the driver presses the brake pedal. 
Specifically, the image is divided into 400 sub-regions each having the same size 
as that of the template. A value calculated by making an appropriate correction 
on arithmetic mean of the intensity values of the pixels included in each 
5 sub-region is set as the intensity value of each sub-region. Thus, one of 
256-level gray scale intensity values (0 to 255) is assigned to each sub -region. 
An initial value for the intensity value is assigned randomly. It should be noted 
that the number of sub-regions in one template is not hmited to 400. The 
number of the sub-regions is chosen to enable optimal control of the primary 

10 vehicle depending on various factors such as processing abiUty of the ECU 74, 
resolution of the camera 102, and size and number of the templates. 

In order to recognize the approach of a front vehicle accurately, the 
template is positioned at the center of the captured image. Figs. 5 and 6 show 
location of a template (indicated by a rectangular black frame) in regard to an 

15 image captured by the camera 102. 

The size of a template may be same as that of the captured image. 
Preferably, the size of the template relative to the image is determined by 
experiments in consideration of various factors such as resolution of the camera 
102, processing abiUty of the ECU 74, number of templates and total number of 

20 sub -regions. 

When the size of a template is too large relative to the image, the template 
becomes prone to be influenced by images specific to each road which have no 
direct relation with the approach of the front vehicle. When the size of a 
template is too small relative to the image, the template may be filled with only 
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the image of the front vehicle and so the approach of the vehicle cannot be 
detected accurately. Therefore, the size of template is preferably determined 
through experiments depending on various factors such as processing ability of 
the ECU 74, resolution of the camera 102, and size and number of the templates. 
5 According to the embodiment, two templates (labeled Mi, M2) are stored in 

the template memory section 116 as shown in Fig. 7. Each template is assigned 
according to optical flows computed based on captured images, which is described 
later. Preparing a plurality of templates enables more accurate determination. 
However, only one template or more than two templates may be prepared. 

10 One feature of the present invention is that the templates Mn are not fixed 

but are updated as needed when the brake pedal is pressed by the driver. When 
the brake pedal is pressed by the driver while driving the primary vehicle, the 
control signal is outputted from the sensor provided to the brake pedal to the 
control signal receiving section 110 in the image processing unit 100. Upon 

15 receiving the control signal, the template update section 114 updates 
corresponding template Mn. This template updating process is described later in 
detail with reference to Fig. 10. 

As described above, instead of preparing many fixed templates in advance, 
a constant number of templates are updated based on braking operation 

20 performed according to personal character of the driver or the traffic conditions. 
Thus the storage capacity for the templates may be reduced. In addition, 
flexible vehicle control according to driving environment and the like may be 
realized. 

It should be noted that the numeric value corresponding to each sub-region 
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in the template is not restricted to the intensity values providing 256-level gray 
scale. The numeric value may be represented in more or fewer level. The 
numeric value may be the value calculated by applying the Fourier transform, 
the Gabor transform, the Wavelet transform to the intensity value. Any 
5 numeric value corresponding to captured image and capable of being matched 
with the image may be used in the invention. 

Now, process of recognizing the approaching vehicle based on the image 
and outputting the control signal to the primary vehicle is described with 
reference to the flowchart in Fig. 8. Hereinafter assume that the template 

10 memory section 116 has two templates Mi and M2 shown in Fig. 7. Weight Wi or 
W2 having opposite polarity is assigned to each sub-region of the two templates 
depending on the direction of optical flows. 

Images captured by the camera 102 in the travehng direction of the 
primary vehicle are sequentially stored in the image memory section 104 (S200). 

15 Each pixel in the images are represented in 8-bit (256-level gray scale) intensity 
values. 

The recognizing section 106 compares the latest image stored in the image 
processing section 104 with all templates Mn in the template memory section 116. 
Then, the recognizing section 106 calculates a similarity factor Sn for each 
20 sub-region in all templates (S202). 

Process for calculating the similarity factor Sn is described with reference 
to Figs. 8 and 9. The recognizing section 106 calculates arithmetic mean of the 
pixel intensity values within a region corresponding to each sub-region of a first 
template Mi for the latest image stored in the image processing section 104 to 
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calculate 400 intensity values having same array with the template. Assume 
that an intensity value of a sub-region in template Mn is Mn(x,y). Intensity 
value after the process in the image captured at time t is represented as It(x,y). 
The coordinates (x,y) indicate the position of a sub-region in the template. For 
5 example, coordinates of a sub-region at the bottom-left corner in the template in 
Fig. 4 is represented as (1,1), and coordinates of a sub-region at the top-right 
corner in the template is represented as (20,20). 

The pixel intensity values of the image calculated as described above and 
the intensity value Mi(x,y) of the template Mi are compared for each sub region. 

10 When the difference between them is smaller than a predetermined value, the 
intensity value of the sub-region is considered to be similar to the intensity value 
of the image. When the difference between them is greater than the 
predetermined value, the intensity value of the sub-region is considered not to be 
similar to the intensity value of the image. 

15 The above-mentioned process may be represented in the following 

equations. The difference Dn(x,y) is represented as^ 



D„(x,y)=M,(x,y)-I,(x,y)| (l) 



20 By comparing the difference Dn(x,y) and the predetermined threshold 

value Gn, the similarity factor Sn may be determined according to the following 
equation. 

. , , Jl (D.(x,y)£G.) 
^■'"■^'no (D.(x.y)>Gj 
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The threshold value Gn is determined in advance from some experiments 
for investigating association between brake timing by the driver and the image at 
that time, so that the accurate vehicle control may be implemented. 

The calculated similarity factor Sn(x,y) has same array with that of the 
5 template. Fig. 9 is one example of the similarity factor Sn(x,y). Based on the 
similarity factor, the degree of similarity between the image and the template 
may be determined. 

Next, the recognizing section 106 uses the latest image It(x,y) and the 
image It- i(x,y) captured immediately before to calculate optical flows in each 
10 sub-region (S204). As is well known, optical flows (vx(x,y), Vy(x,y)) may be 
calculated from the following equation. 

axv, (x, y)-^dyv/x,y) + 8t = 0 (2) 

15 (x,y) are coordinates designating each sub-region within the array as 

described above. Since the primary vehicle using the vehicle control system 80 
normally travels forward / backward or turn left / right, optical flows in y 
direction (vertical direction) is much smaller than that in x direction (horizontal 
direction). Therefore, only x component Vx(x,y) is calculated with the equation 

20 (2) in order to reduce computing load in this embodiment. 

Based on the similarity factor Sn(x,y) and the optical flow Vx(x,y), the 
recognizing section 106 calculates local movement Vn(x,y) in each sub-region 
according to the following equation (S206). 
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V,(x,y) = S,(x,y).v,(x,y) (3) 

The local movement Vn(x,y) reflects the value of the optical flows of the 
sub-region having similarity factor Sn(x,y) in Fig. 9. It should be noted that the 
5 local movement Vn(x,y) includes directional information for the direction of the 
optical flows in each sub-region. 

Processes from step S200 to step S206 is also performed on all sub-regions 
within the second template M2. 

The recognizing section 106 then calculates an evaluation value B by the 

10 following equation. 

B = Z a n E Z W„ (X, y) V„ (X, y) (4) 

n X y 



where Wn(x,y) represents a weight for each sub -region. As shown in Fig. 7, a 
15 positive weight Wi is assigned to each sub-region in the template Mi and a 
negative weight W2 is assigned to each sub-region in the template M2. Therefore, 
assuming that the leftward direction in the templates is "positive" and the 
rightward direction in the templates is "negative", the larger the local movement 
Vn(x,y) in the leftward direction in the template Mi is, and also the larger the 
20 local movement Vn(x,y) in the rightward direction of the template M2 is, the 
larger the evaluation value B becomes. On represents a template coefficient for 
each template Mn and 0<a„ <1. "n" is a total number of templates being 
stored and set to n=2, for example. Weight Wn(x,y) is updated as needed when 
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the brake pedal is pressed by the driver. This process will be described later 
with reference to Fig. 10. The template coefficient an is pre-determined for each 
template by experiments or so in order to perform the vehicle control accurately. 
Typically, the closer the vehicle ahead approaches to the primary vehicle, 
5 the greater the optical flow Vx(x,y) becomes. The evaluation value B is 
calculated by using the weight Wn(x,y) to strengthen the local movement Vn(x,y), 
and further by adding this with respect to all the templates. Therefore, the 
more sub-regions have similarity factor Sn(x,y)=l, the larger the optical flow 
Vx(x,y) becomes, the larger the weight Wn(x,y) becomes, and the greater the 

10 evaluation value B becomes. In other words, the more similar the captured 
image is to the image previously captured when the brake pedal is pressed, the 
greater the evaluation value B becomes. The evaluation value B becomes larger 
when the approaching speed to the front vehicle is faster. 

The recognizing section 106 determines whether the evaluation value B 

15 calculated by equation (4) is greater than a predetermined threshold value ki 
(S210). When the evaluation value B is smaller than the threshold value ki, this 
routine is finished. When the evaluation value B is greater than the threshold 
value ki, the recognizing section 106 concludes that collision avoidance action (for 
example, braking) is required. Then, the recognizing section 106 determines 

20 whether the driver actually takes the collision avoidance action based on 
presence/absence of a signal firom the sensor installed to the brake pedal or 
accelerator 4 (S212). When the driver takes the collision avoidance action, this 
routine is finished. When the driver does not take the collision avoidance action, 
the instruction section 108 sends control instruction to the ECU 74 (S212). In 
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response to the control instruction, the ECU 74 executes some operation such as 
closing the throttle valve 76. Thus, since the driving force of the primary vehicle 
decreases and sufficient negative pressure is supplied to the brake booster 6, 
sufficient braking force is provided surely under harsh braking, leading to 
5 improve safety of the primary vehicle. Simultaneously with or instead of the 
close of the throttle valve 76, an alarm or warning display may be generated on 
the vehicle. Alternatively, after determining as "NO" at step S212, the control 
instruction on step S214 may be send with some delay depending on the 
approaching speed to the front vehicle. 

10 In comparing step of the evaluation value B at S210, when the threshold 

value ki is set at too low value, the throttle valve 76 is closed at early timing. 
This may result in decreasing the driving force of the primary vehicle 
unnecessarily. When the threshold value ki is set at too high, the vehicle may 
not be decelerated in case of necessity. Therefore, the threshold value ki should 

15 be set at appropriate value considering various parameters such as the 
environment surrounding the primary vehicle, velocity of the primary vehicle, 
computing performance of the image processing unit 100 and layout and number 
of templates Mn. 

According to another embodiment, the recognizing section 106 has a 
20 plurality of threshold values (for example, ki, k2 and ks) for comparing step at 
S210 and executes a different vehicle control depending on which threshold value 
the evaluation value B exceeds. For example, when the evaluation value B 
exceeds threshold value ki, the throttle valve 76 is controlled to be closed. When 
the evaluation value B exceeds k2 (>ki), an alarm or warning display is issued to 
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the driver. When the evaluation value B exceeds ka (>k2), a braking pedal is 
pressed forcibly. 

According to further another embodiment, instead of providing step S212 
in Fig. 8, a control instruction may be issued when B>ki at step S210 regardless 
5 of the presence or absence of the collision avoidance action. 

The number of templates is not restricted at two in the process in Fig. 8. 
For example, since there are some differences in the optical flows according to the 
type of movements (traveling straight traveling, turning right or left), different 
template set (shown in Fig. 7) may be prepared for each type of movement. In 
10 this case, appropriate template set is selected based on the angle of a steering 
wheel. 

Alternatively, various template sets may be prepared depending on the 
type of a front vehicle (large or small, etc.) or its road surroundings. In this case, 
appropriate template set is selected depending on the front vehicle and the 
15 surroundings. For example, three template sets, one for freeways, one for urban 
areas and one for mountain roads, may be prepared. Depending on the current 
road surroundings provided by a car navigation system, an appropriate template 
set may be selected for use in the process in Fig. 8. 

With reference to Fig. 10, updating process for intensity value Mn(x,y) and 
20 weight Wn(x,y) in each sub-region of the template Mn is described. This process 
is performed on all sub-regions (for example, 400 sub-regions) in all templates 
(for example, two templates) every time the driver press the brake pedal. 

The control signal receiving section 110 determines whether the brake 
pedal is pressed by the driver based on the signal from the sensor provided to the 
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brake pedal 2 (S300). 

When the brake pedal is not pressed, the template update section 114 then 
determines whether an absolute value |V^ (x, y)| of the current local movement is 
greater than a predetermined threshold value "d" (S302). If jV„(x,y)|<d, this 
5 routine is finished because it is unnecessary to update the pixel intensity value 
and the weight Wn for the sub-region in the template. 

If I (x, y)| > d , it is considered that the local movement Vn(x,y) of the 

sub-region is increased even though the brake pedal is not pressed. In this case, 
the template update section 114 judges that unnecessary information for judging 
10 approach of the vehicle is reflected in the image and then sets the weight Wn(x,y) 
of that sub-region to zero (S304). By this process, information related to the 
sub-region having the local movement Vn(x,y) greater than the threshold value "d 
is canceled. 

When the brake pedal is pressed at step S300, the template update section 
15 114 determines whether |Vn(x,y)|>d and |Wn(x,y)l>0 is established (S306). 

When these conditions are established, the sub-region is considered to include 

the necessary information for detecting the approach of the front vehicle. 

Therefore, the template update section 114 updates the intensity value Mn(x,y) 

and the weight Wn(x,y) for the sub-region (S308). 
20 The intensity value Mn(x,y) and the weight Wn(x,y) are updated by the 

following equations- 



(x, y) = M„ (x, y) - q(M„ (x, y) - 1, (x, y)) 
W„(x,y) = W„(x,y) + w 



(5) 
(6) 
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where "q" is a fixed value within the range of 0 < q < 1 , and "w" is an initial value 
of the weight Wn(x,y). "w" in the equation (6) is a negative value for template M2. 
By using these equations (5) and (6), the information related to the approach of 
5 the vehicle is emphasized, among the information included in the template, 

When I Vn(x,y) I >d and I Wn(x,y) I >0 are not established at step S306, the 
template update section 114 determines whether the weight Wn(x,y) equals to 
zero (S310). When Wn(x,y) is not zero, this routine is finished. When Wn(x,y) is 
zero, the template update section 114 then sets the intensity value of the 

10 corresponding inputted image It as the intensity value of the template Mn of the 
sub-region and assigns the initial value "w" to the weight Wn(x,y) (S3 12). 
When the weight Wn(x,y) is still zero regardless of the image which is necessary 
for detecting the approach of the vehicle ahead has been captured in the travehng 
direction of the primary vehicle, the intensity value of the corresponding 

15 sub-region of the inputted image It is set to the sub-region because it is judged 
that this is the first time that the image fi-om that part was obtained. 

Now, the process in Fig. 10 is explained in detail with reference to Figs. 5 
and 6. Figs. 5 and 6 are examples of the images captured by the camera 102 in 
the travehng direction of the primary vehicle. These images are different in 

20 distance between vehicles. Templates, each of which is made firom the portion of 
the image indicated by a rectangular black frame in the captured image, are 
considered to be different because positional relationship between the vehicle and 
background is different in these images. Assume that the driver presses the 
brake pedal when the distance between the vehicles almost equals to the distance 
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shown in the image in Fig. 5. Then, when an image having high similarity level 
with the template of the image in Fig. 5 is captured, it can be judged that the 
collision avoidance action should be performed. Then, the template update 
section 114 updates the templates according to the equations (5) and (6) so that 
5 the image captured when the brake pedal is pressed is reflected to a current 
template. Thus, the templates are updated repeatedly based on the image 
captured every time the driver presses the brake pedal. 

More similar the template is to the image It when the brake pedal is 
pressed, more times the initial value w is added to of the sub-region in the 
10 template Mn. Thus, absolute value of the weight Wn(x,y) becomes larger. 
Accordingly, when the braking is repeated many times every time a similar image 
is captured, sub regions having the similarity factor Sn(x,y)=l is selected in the 
image It and these sub-regions are enhanced (extracted) by the weight Wn(x,y). 
On the other hand, information for sub -regions not consistently detected (for 
15 example, an object which appears irregularly in front of the primary vehicle) is 
dropped. Thus, by the process of Fig. 10, selection of information is executed by 
enhancing information having high reliability (importance) for judging the 
approach of the front vehicle and deleting other information. 

When the brake pedal is kept being pressed, the templates may be updated 
20 at the time when the brake pedal is first pressed down. When the brake pedal is 
pressed more deeply, the templates may be updated one more time. 

Note that the threshold value "d", the coefficient "q", and the weighted 
initial value "w" used in the process in Fig. 10 should be selected appropriately 
depending on various factors such as computing performance of the ECU 74, 
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resolution of the camera 102, the number of templates or sub-regions so that the 
template Mn and the weight Wn are updated to detect the approach of the vehicle 
more precisely. As with the process in Fig. 8, the number of template set in Fig. 
10 is not limited to one. A plurality of templates may be prepared corresponding 
5 to the direction of movement of the primary vehicle and one template set may be 
selected for updating depending on the angle of a steering wheel. Alternatively, 
As with the process in Fig. 8, one template set may be selected for updating 
among a plurality of template sets depending on the type of a front vehicle and 
surroundings. 

10 In conclusion, the vehicle control system according to the invention 

operates as described below. When the primary vehicle is driving, templates are 
updated to reflect features of an image captured every time a driver presses a 
brake pedal if some conditions are satisfied. In parallel with this process, the 
vehicle control system determines whether the primary vehicle needs to be 

15 decelerated or halted because another vehicle driving ahead approaches to the 
primary vehicle based on the image captured in the traveUng direction of the 
primary vehicle. Then, when the primary vehicle needs to be decelerated or 
halted and when the driver performs no avoidance action such as releasing a 
accelerator or pressing the brake pedal, the vehicle control system controls the 

20 throttle valve 76 of the brake system 1 to close it. As a result, the driving force 
of the primary vehicle is decreased and sufficient negative pressure is supplied to 
the brake booster 6. Therefore, even when the driver's judgment is delayed and 
the harsh braking is performed, sufficient vehicle braking force may be appUed to 
the primary vehicle. In addition, since the throttle valve is forcibly closed. 
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driving force of the primary vehicle is suppressed even when the accelerator 
pedal keeps being depressed. It can be expected that the vehicle control system 
according to the invention have the excellent effect when distance between 
vehicles is relatively small (such as when driving in traffic jam or on a city road 
and when the brake pedal is pressed frequently) because the driver's attention 
tends to be distracted from driving under such traffic conditions. 

Although some preferred embodiments have been described with reference 
to drawings, it is not intended to limit the scope of the invention to such 
embodiments. For example, instead of closing the throttle valve 76, the vehicle 
control system may decelerate or brake the primary vehicle directly without 
waiting for the response by the driver, or simply issue an alarm to the driver. 
Images captured when a driver presses an accelerator may be stored as 
templates. In this case, when the distance between vehicles gets wider, the 
vehicle control system may accelerate the primary vehicle. Further, detection of 
approach of the vehicle is not limited for front vehicle but for side vehicle or back 
vehicle by modifying an installation position of a camera and calculation of an 
evaluation value B. The vehicle control system according to the invention may 
also be applied to a two-wheel vehicle instead of a four-wheel vehicle. 

According to the invention, since a template update means is provided for 
updating templates every time a brake pedal is pressed by the driver, a vehicle 
control system may be realized with relatively small storage capacity for flexibly 
adapting with the personal character of the driver and the conditions 
surrounding the primary vehicle. 



